{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb2b6903",
   "metadata": {},
   "source": [
    "# Managing a local MegaDetector batch\n",
    "\n",
    "This notebook represents an interactive process for running MegaDetector on large batches of images, including typical and optional postprocessing steps.  Everything after \"Merge results...\" is basically optional, and we typically do a mix of these optional steps, depending on the job.\n",
    "\n",
    "This notebook is auto-generated from manage_local_batch.py (a cell-delimited .py file that is used the same way, typically in Spyder or VS Code).\n",
    "\n",
    " Semi-automated process for managing a local MegaDetector job, including\n",
    " standard postprocessing steps.\n",
    "\n",
    "\n",
    " This script is not intended to be run from top to bottom like a typical Python script,\n",
    " it's a notebook disguised with a .py extension.  It's the Bestest Most Awesome way to\n",
    " run MegaDetector, but it's also pretty subtle; if you want to play with this, you might\n",
    " want to check in with cameratraps@lila.science for some tips.  Otherwise... YMMV.\n",
    "\n",
    "\n",
    " Some general notes on using this script, which I run in Spyder, though everything will be\n",
    " the same if you are reading this in Jupyter Notebook (using the .ipynb version of the \n",
    " script):\n",
    "\n",
    "\n",
    " * Typically when I have a MegaDetector job to run, I make a copy of this script.  Let's \n",
    "   say I'm running a job for an organization called \"bibblebop\"; I have a big folder of\n",
    "   job-specific copies of this script, and I might save a new one called \"bibblebop-2023-07-26.py\" \n",
    "   (the filename doesn't matter, it just helps me keep these organized).\n",
    "\n",
    "\n",
    " * There are three variables you need to set in this script before you start running code:\n",
    "   \"input_path\", \"organization_name_short\", and \"job_date\".  You will get a sensible error if you forget \n",
    "   to set any of these.  In this case I might set those to \"/data/bibblebobcamerastuff\",\n",
    "   \"bibblebop\", and \"2023-07-26\", respectively.\n",
    "\n",
    "\n",
    " * The defaults assume you want to split the job into two tasks (this is the default because I have \n",
    "   two GPUs).  Nothing bad will happen if you do this on a zero-GPU or single-GPU machine, but if you\n",
    "   want everything to run in one logical task, change \"n_gpus\" and \"n_jobs\" to 1 (instead of 2).\n",
    "\n",
    "\n",
    " * After setting the required variables, I run the first few cells - up to and including the one \n",
    "   called \"Generate commands\" - which collectively take basically zero seconds.  After you run the\n",
    "   \"Generate commands\" cell, you will have a folder that looks something like:\n",
    "\n",
    "\n",
    "     ~/postprocessing/bibblebop/bibblebop-2023-07-06-mdv5a/\n",
    "\n",
    "\n",
    "   On Windows, this means:\n",
    "\n",
    "\n",
    "     ~/postprocessing/bibblebop/bibblebop-2023-07-06-mdv5a/\n",
    "\n",
    "\n",
    "   Everything related to this job - scripts, outputs, intermediate stuff - will be in this folder.\n",
    "   Specifically, after the \"Generate commands\" cell, you'll have scripts in that folder called something\n",
    "   like:\n",
    "\n",
    "\n",
    "   run_chunk_000_gpu_00.sh (or .bat on Windows)\n",
    "\n",
    "\n",
    "   Personally, I like to run that script directly in a command prompt (I just leave Spyder open, though \n",
    "   it's OK if Spyder gets shut down while MD is running).\n",
    "\n",
    "\n",
    "   At this point, once you get the hang of it, you've invested about zero seconds of human time,\n",
    "   but possibly several days of unattended compute time, depending on the size of your job.\n",
    "\n",
    "\n",
    " * Then when the jobs are done, back to the interactive environment!  I run the next few cells,\n",
    "   which make sure the job finished OK, and the cell called \"Post-processing (pre-RDE)\", which \n",
    "   generates an HTML preview of the results.  You are very plausibly done at this point, and can ignore\n",
    "   all the remaining cells.  If you want to do things like repeat detection elimination, or running \n",
    "   a classifier, or splitting your results file up in specialized ways, there are cells for all of those\n",
    "   things, but now you're in power-user territory, so I'm going to leave this guide here.  Email\n",
    "   cameratraps@lila.science with questions about the fancy stuff.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "883ac407",
   "metadata": {},
   "source": [
    "## Imports and constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2a70428",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import stat\n",
    "import time\n",
    "import re\n",
    "\n",
    "import humanfriendly\n",
    "\n",
    "from tqdm import tqdm\n",
    "from collections import defaultdict\n",
    "\n",
    "from megadetector.utils import path_utils\n",
    "from megadetector.utils.ct_utils import split_list_into_n_chunks\n",
    "from megadetector.utils.ct_utils import image_file_to_camera_folder\n",
    "from megadetector.detection.run_detector_batch import \\\n",
    "    load_and_run_detector_batch, write_results_to_file\n",
    "from megadetector.detection.run_detector import DEFAULT_OUTPUT_CONFIDENCE_THRESHOLD\n",
    "from megadetector.detection.run_detector import estimate_md_images_per_second\n",
    "from megadetector.postprocessing.postprocess_batch_results import \\\n",
    "    PostProcessingOptions, process_batch_results\n",
    "from megadetector.detection.run_detector import get_detector_version_from_filename\n",
    "\n",
    "## Inference options\n",
    "\n",
    "# To specify a non-default confidence threshold for including detections in the .json file\n",
    "json_threshold = None\n",
    "\n",
    "# Turn warnings into errors if more than this many images are missing\n",
    "max_tolerable_failed_images = 100\n",
    "\n",
    "# Should we supply the --image_queue_option to run_detector_batch.py?  I only set this\n",
    "# when I have a very slow drive and a comparably fast GPU.  When this is enabled, checkpointing\n",
    "# is not supported within a job, so I set n_jobs to a large number (typically 100).\n",
    "use_image_queue = False\n",
    "\n",
    "# Only relevant when we're using a single GPU\n",
    "default_gpu_number = 0\n",
    "\n",
    "# Should we supply --quiet to run_detector_batch.py?\n",
    "quiet_mode = True\n",
    "\n",
    "# Specify a target image size when running MD... strongly recommended to leave this at \"None\"\n",
    "#\n",
    "# When using augmented inference, if you leave this at \"None\", run_inference_with_yolov5_val.py\n",
    "# will use its default size, which is 1280 * 1.3, which is almost always what you want.\n",
    "image_size = None\n",
    "\n",
    "# Should we include image size, timestamp, and/or EXIF data in MD output?\n",
    "include_image_size = False\n",
    "include_image_timestamp = False\n",
    "include_exif_data = False\n",
    "\n",
    "# Only relevant when running on CPU\n",
    "ncores = 1\n",
    "\n",
    "# If False, we'll load chunk files with file lists if they exist\n",
    "force_enumeration = False\n",
    "\n",
    "# Prefer threads on Windows, processes on Linux\n",
    "parallelization_defaults_to_threads = False\n",
    "\n",
    "# This is for things like image rendering, not for MegaDetector\n",
    "default_workers_for_parallel_tasks = 30\n",
    "\n",
    "overwrite_handling = 'skip' # 'skip', 'error', or 'overwrite'\n",
    "\n",
    "# The function used to get camera names from image paths; can also replace this\n",
    "# with a custom function.\n",
    "relative_path_to_location = image_file_to_camera_folder\n",
    "\n",
    "# This will be the .json results file after RDE; if this is still None when\n",
    "# we get to classification stuff, that will indicate that we didn't do RDE.\n",
    "filtered_output_filename = None\n",
    "\n",
    "\n",
    "# OS-specific script line continuation character (modified later if we're running on Windows)\n",
    "slcc = '\\\\'\n",
    "\n",
    "# OS-specific script comment character (modified later if we're running on Windows)\n",
    "scc = '#'\n",
    "\n",
    "# # OS-specific script extension (modified later if we're running on Windows)\n",
    "script_extension = '.sh'\n",
    "\n",
    "# Stuff we stick into scripts to ensure early termination if there's an error\n",
    "script_header = '#!/bin/bash\\n\\nset -e\\n'\n",
    "\n",
    "# Include this after each command in a .sh/.bat file\n",
    "command_suffix = ''\n",
    "\n",
    "if os.name == 'nt':\n",
    "\n",
    "    script_header = ''\n",
    "    slcc = '^'\n",
    "    scc = 'REM'\n",
    "    script_extension = '.bat'\n",
    "\n",
    "    command_suffix = 'if %errorlevel% neq 0 exit /b %errorlevel%\\n'\n",
    "\n",
    "    # My experience has been that Python multiprocessing is flaky on Windows, so\n",
    "    # default to threads on Windows\n",
    "    parallelization_defaults_to_threads = True\n",
    "    default_workers_for_parallel_tasks = 10\n",
    "\n",
    "\n",
    "## Constants related to using YOLOv5's val.py\n",
    "\n",
    "# Should we use YOLOv5's val.py instead of run_detector_batch.py?\n",
    "use_yolo_inference_scripts = False\n",
    "\n",
    "# Directory in which to run val.py (relevant for YOLOv5, not for YOLOv8)\n",
    "yolo_working_dir = os.path.expanduser('~/git/yolov5')\n",
    "\n",
    "# Only used for loading the mapping from class indices to names\n",
    "yolo_dataset_file = None\n",
    "\n",
    "# 'yolov5' or 'yolov8'; assumes YOLOv5 if this is None\n",
    "yolo_model_type = None\n",
    "\n",
    "# inference batch size\n",
    "yolo_batch_size = 1\n",
    "\n",
    "# Should we remove intermediate files used for running YOLOv5's val.py?\n",
    "#\n",
    "# Only relevant if use_yolo_inference_scripts is True.\n",
    "remove_yolo_intermediate_results = True\n",
    "remove_yolo_symlink_folder = True\n",
    "use_symlinks_for_yolo_inference = True\n",
    "write_yolo_debug_output = False\n",
    "\n",
    "# Should we apply YOLOv5's test-time augmentation?\n",
    "augment = False\n",
    "\n",
    "\n",
    "## Constants related to tiled inference\n",
    "\n",
    "use_tiled_inference = False\n",
    "\n",
    "# Should we delete tiles after each job?  Only set this to False for debugging;\n",
    "# large jobs will take up a lot of space if you keep tiles around after each task.\n",
    "remove_tiles = True\n",
    "tile_size = (1280,1280)\n",
    "tile_overlap = 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "795f4cf2",
   "metadata": {},
   "source": [
    "## Constants I set per script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67177249",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_path = '/drive/organization'\n",
    "\n",
    "assert not (input_path.endswith('/') or input_path.endswith('\\\\'))\n",
    "assert os.path.isdir(input_path), 'Could not find input folder {}'.format(input_path)\n",
    "input_path = input_path.replace('\\\\','/')\n",
    "\n",
    "organization_name_short = 'organization'\n",
    "job_date = None # '2024-01-01'\n",
    "assert job_date is not None and organization_name_short != 'organization'\n",
    "\n",
    "# Optional descriptor\n",
    "job_tag = None\n",
    "\n",
    "if job_tag is None:\n",
    "    job_description_string = ''\n",
    "else:\n",
    "    job_description_string = '-' + job_tag\n",
    "\n",
    "model_file = 'MDV5A' # 'MDV5A', 'MDV5B', 'MDV4'\n",
    "\n",
    "postprocessing_base = os.path.expanduser('~/postprocessing')\n",
    "\n",
    "# Number of jobs to split data into, typically equal to the number of available GPUs, though\n",
    "# when using augmentation or an image queue (and thus not using checkpoints), I typically\n",
    "# use ~100 jobs per GPU; those serve as de facto checkpoints.\n",
    "n_jobs = 2\n",
    "n_gpus = 2\n",
    "\n",
    "# Set to \"None\" when using augmentation or an image queue, which don't currently support\n",
    "# checkpointing.  Don't worry, this will be assert()'d in the next cell.\n",
    "checkpoint_frequency = 10000\n",
    "\n",
    "# Estimate inference speed for the current GPU\n",
    "approx_images_per_second = estimate_md_images_per_second(model_file)\n",
    "\n",
    "# Rough estimate for the inference time cost of augmentation\n",
    "if augment and (approx_images_per_second is not None):\n",
    "    approx_images_per_second = approx_images_per_second * 0.7\n",
    "\n",
    "base_task_name = organization_name_short + '-' + job_date + job_description_string + '-' + \\\n",
    "    get_detector_version_from_filename(model_file)\n",
    "base_output_folder_name = os.path.join(postprocessing_base,organization_name_short)\n",
    "os.makedirs(base_output_folder_name,exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15968fee",
   "metadata": {},
   "source": [
    "## Derived variables, constant validation, path setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e5a399",
   "metadata": {},
   "outputs": [],
   "source": [
    "if use_image_queue:\n",
    "    assert checkpoint_frequency is None,\\\n",
    "        'Checkpointing is not supported when using an image queue'\n",
    "\n",
    "if augment:\n",
    "    assert checkpoint_frequency is None,\\\n",
    "        'Checkpointing is not supported when using augmentation'\n",
    "\n",
    "    assert use_yolo_inference_scripts,\\\n",
    "        'Augmentation is only supported when running with the YOLO inference scripts'\n",
    "\n",
    "if use_tiled_inference:\n",
    "    assert not augment, \\\n",
    "        'Augmentation is not supported when using tiled inference'\n",
    "    assert not use_yolo_inference_scripts, \\\n",
    "        'Using the YOLO inference script is not supported when using tiled inference'\n",
    "    assert checkpoint_frequency is None, \\\n",
    "        'Checkpointing is not supported when using tiled inference'\n",
    "\n",
    "filename_base = os.path.join(base_output_folder_name, base_task_name)\n",
    "combined_api_output_folder = os.path.join(filename_base, 'combined_api_outputs')\n",
    "postprocessing_output_folder = os.path.join(filename_base, 'preview')\n",
    "\n",
    "combined_api_output_file = os.path.join(\n",
    "    combined_api_output_folder,\n",
    "    '{}_detections.json'.format(base_task_name))\n",
    "\n",
    "os.makedirs(filename_base, exist_ok=True)\n",
    "os.makedirs(combined_api_output_folder, exist_ok=True)\n",
    "os.makedirs(postprocessing_output_folder, exist_ok=True)\n",
    "\n",
    "if input_path.endswith('/'):\n",
    "    input_path = input_path[0:-1]\n",
    "\n",
    "print('Output folder:\\n{}'.format(filename_base))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beaf5baf",
   "metadata": {},
   "source": [
    "## Enumerate files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "280be396",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Have we already listed files for this job?\n",
    "chunk_files = os.listdir(filename_base)\n",
    "pattern = re.compile('chunk\\d+.json')\n",
    "chunk_files = [fn for fn in chunk_files if pattern.match(fn)]\n",
    "\n",
    "if (not force_enumeration) and (len(chunk_files) > 0):\n",
    "\n",
    "    print('Found {} chunk files in folder {}, bypassing enumeration'.format(\n",
    "        len(chunk_files),\n",
    "        filename_base))\n",
    "\n",
    "    all_images = []\n",
    "    for fn in chunk_files:\n",
    "        with open(os.path.join(filename_base,fn),'r') as f:\n",
    "            chunk = json.load(f)\n",
    "            assert isinstance(chunk,list)\n",
    "            all_images.extend(chunk)\n",
    "    all_images = sorted(all_images)\n",
    "\n",
    "    print('Loaded {} image files from {} chunks in {}'.format(\n",
    "        len(all_images),len(chunk_files),filename_base))\n",
    "\n",
    "else:\n",
    "\n",
    "    print('Enumerating image files in {}'.format(input_path))\n",
    "\n",
    "    all_images = sorted(path_utils.find_images(input_path,recursive=True,convert_slashes=True))\n",
    "\n",
    "    # It's common to run this notebook on an external drive with the main folders in the drive root\n",
    "    all_images = [fn for fn in all_images if not \\\n",
    "                  (fn.startswith('$RECYCLE') or fn.startswith('System Volume Information'))]\n",
    "\n",
    "    print('')\n",
    "\n",
    "    print('Enumerated {} image files in {}'.format(len(all_images),input_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90606ec0",
   "metadata": {},
   "source": [
    "## Divide images into chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cf5b8d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "folder_chunks = split_list_into_n_chunks(all_images,n_jobs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbf86eb",
   "metadata": {},
   "source": [
    "## Estimate total time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "931f56f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "if approx_images_per_second is None:\n",
    "\n",
    "    print(\"Can't estimate inference time for the current environment\")\n",
    "\n",
    "else:\n",
    "\n",
    "    n_images = len(all_images)\n",
    "    execution_seconds = n_images / approx_images_per_second\n",
    "    wallclock_seconds = execution_seconds / n_gpus\n",
    "    print('Expected time: {}'.format(humanfriendly.format_timespan(wallclock_seconds)))\n",
    "\n",
    "    seconds_per_chunk = len(folder_chunks[0]) / approx_images_per_second\n",
    "    print('Expected time per chunk: {}'.format(humanfriendly.format_timespan(seconds_per_chunk)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bad6f800",
   "metadata": {},
   "source": [
    "## Write file lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa30d095",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_info = []\n",
    "\n",
    "for i_chunk,chunk_list in enumerate(folder_chunks):\n",
    "\n",
    "    chunk_fn = os.path.join(filename_base,'chunk{}.json'.format(str(i_chunk).zfill(3)))\n",
    "    task_info.append({'id':i_chunk,'input_file':chunk_fn})\n",
    "    path_utils.write_list_to_file(chunk_fn, chunk_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4a5a99a",
   "metadata": {},
   "source": [
    "## Generate commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2933e219",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A list of the scripts tied to each GPU, as absolute paths.  We'll write this out at\n",
    "# the end so each GPU's list of commands can be run at once\n",
    "gpu_to_scripts = defaultdict(list)\n",
    "\n",
    "# i_task = 0; task = task_info[i_task]\n",
    "for i_task,task in enumerate(task_info):\n",
    "\n",
    "    chunk_file = task['input_file']\n",
    "    checkpoint_filename = chunk_file.replace('.json','_checkpoint.json')\n",
    "\n",
    "    output_fn = chunk_file.replace('.json','_results.json')\n",
    "\n",
    "    task['output_file'] = output_fn\n",
    "\n",
    "    if n_gpus > 1:\n",
    "        gpu_number = i_task % n_gpus\n",
    "    else:\n",
    "        gpu_number = default_gpu_number\n",
    "\n",
    "    image_size_string = ''\n",
    "    if image_size is not None:\n",
    "        image_size_string = '--image_size {}'.format(image_size)\n",
    "\n",
    "    # Generate the script to run MD\n",
    "\n",
    "    if use_yolo_inference_scripts:\n",
    "\n",
    "        augment_string = ''\n",
    "        if augment:\n",
    "            augment_string = '--augment_enabled 1'\n",
    "        else:\n",
    "            augment_string = '--augment_enabled 0'\n",
    "\n",
    "        batch_string = '--batch_size {}'.format(yolo_batch_size)\n",
    "\n",
    "        symlink_folder = os.path.join(filename_base,'symlinks','symlinks_{}'.format(\n",
    "            str(i_task).zfill(3)))\n",
    "        yolo_results_folder = os.path.join(filename_base,'yolo_results','yolo_results_{}'.format(\n",
    "            str(i_task).zfill(3)))\n",
    "\n",
    "        symlink_folder_string = '--symlink_folder \"{}\"'.format(symlink_folder)\n",
    "        yolo_results_folder_string = '--yolo_results_folder \"{}\"'.format(yolo_results_folder)\n",
    "\n",
    "        remove_symlink_folder_string = ''\n",
    "        if not remove_yolo_symlink_folder:\n",
    "            remove_symlink_folder_string = '--no_remove_symlink_folder'\n",
    "\n",
    "        write_yolo_debug_output_string = ''\n",
    "        if write_yolo_debug_output:\n",
    "            write_yolo_debug_output = '--write_yolo_debug_output'\n",
    "\n",
    "        remove_yolo_results_string = ''\n",
    "        if not remove_yolo_intermediate_results:\n",
    "            remove_yolo_results_string = '--no_remove_yolo_results_folder'\n",
    "\n",
    "        confidence_threshold_string = ''\n",
    "        if json_threshold is not None:\n",
    "            confidence_threshold_string = '--conf_thres {}'.format(json_threshold)\n",
    "        else:\n",
    "            confidence_threshold_string = '--conf_thres {}'.format(DEFAULT_OUTPUT_CONFIDENCE_THRESHOLD)\n",
    "\n",
    "        cmd = ''\n",
    "\n",
    "        device_string = '--device {}'.format(gpu_number)\n",
    "\n",
    "        overwrite_handling_string = '--overwrite_handling {}'.format(overwrite_handling)\n",
    "\n",
    "        cmd += f'python run_inference_with_yolov5_val.py \"{model_file}\" \"{chunk_file}\" \"{output_fn}\" '\n",
    "        cmd += f'{image_size_string} {augment_string} '\n",
    "        cmd += f'{symlink_folder_string} {yolo_results_folder_string} {remove_yolo_results_string} '\n",
    "        cmd += f'{remove_symlink_folder_string} {confidence_threshold_string} {device_string} '\n",
    "        cmd += f'{overwrite_handling_string} {batch_string} {write_yolo_debug_output_string}'\n",
    "\n",
    "        if yolo_working_dir is not None:\n",
    "            cmd += f' --yolo_working_folder \"{yolo_working_dir}\"'\n",
    "        if yolo_dataset_file is not None:\n",
    "            cmd += ' --yolo_dataset_file \"{}\"'.format(yolo_dataset_file)\n",
    "        if yolo_model_type is not None:\n",
    "            cmd += ' --model_type {}'.format(yolo_model_type)\n",
    "\n",
    "        if not use_symlinks_for_yolo_inference:\n",
    "            cmd += ' --no_use_symlinks'\n",
    "\n",
    "        cmd += '\\n'\n",
    "\n",
    "    elif use_tiled_inference:\n",
    "\n",
    "        tiling_folder = os.path.join(filename_base,'tile_cache','tile_cache_{}'.format(\n",
    "            str(i_task).zfill(3)))\n",
    "\n",
    "        if os.name == 'nt':\n",
    "            cuda_string = f'set CUDA_VISIBLE_DEVICES={gpu_number} & '\n",
    "        else:\n",
    "            cuda_string = f'CUDA_VISIBLE_DEVICES={gpu_number} '\n",
    "\n",
    "        cmd = f'{cuda_string} python run_tiled_inference.py \"{model_file}\" \"{input_path}\" \"{tiling_folder}\" \"{output_fn}\"'\n",
    "\n",
    "        cmd += f' --image_list \"{chunk_file}\"'\n",
    "        cmd += f' --overwrite_handling {overwrite_handling}'\n",
    "\n",
    "        if not remove_tiles:\n",
    "            cmd += ' --no_remove_tiles'\n",
    "\n",
    "        # If we're using non-default tile sizes\n",
    "        if tile_size is not None and (tile_size[0] > 0 or tile_size[1] > 0):\n",
    "            cmd += ' --tile_size_x {} --tile_size_y {}'.format(tile_size[0],tile_size[1])\n",
    "\n",
    "        if tile_overlap is not None:\n",
    "            cmd += f' --tile_overlap {tile_overlap}'\n",
    "\n",
    "    else:\n",
    "\n",
    "        if os.name == 'nt':\n",
    "            cuda_string = f'set CUDA_VISIBLE_DEVICES={gpu_number} & '\n",
    "        else:\n",
    "            cuda_string = f'CUDA_VISIBLE_DEVICES={gpu_number} '\n",
    "\n",
    "        checkpoint_frequency_string = ''\n",
    "        checkpoint_path_string = ''\n",
    "\n",
    "        if checkpoint_frequency is not None and checkpoint_frequency > 0:\n",
    "            checkpoint_frequency_string = f'--checkpoint_frequency {checkpoint_frequency}'\n",
    "            checkpoint_path_string = '--checkpoint_path \"{}\"'.format(checkpoint_filename)\n",
    "\n",
    "        use_image_queue_string = ''\n",
    "        if (use_image_queue):\n",
    "            use_image_queue_string = '--use_image_queue'\n",
    "\n",
    "        ncores_string = ''\n",
    "        if (ncores > 1):\n",
    "            ncores_string = '--ncores {}'.format(ncores)\n",
    "\n",
    "        quiet_string = ''\n",
    "        if quiet_mode:\n",
    "            quiet_string = '--quiet'\n",
    "\n",
    "        confidence_threshold_string = ''\n",
    "        if json_threshold is not None:\n",
    "            confidence_threshold_string = '--threshold {}'.format(json_threshold)\n",
    "\n",
    "        overwrite_handling_string = '--overwrite_handling {}'.format(overwrite_handling)\n",
    "        cmd = f'{cuda_string} python run_detector_batch.py \"{model_file}\" \"{chunk_file}\" \"{output_fn}\" {checkpoint_frequency_string} {checkpoint_path_string} {use_image_queue_string} {ncores_string} {quiet_string} {image_size_string} {confidence_threshold_string} {overwrite_handling_string}'\n",
    "\n",
    "        if include_image_size:\n",
    "            cmd += ' --include_image_size'\n",
    "        if include_image_timestamp:\n",
    "            cmd += ' --include_image_timestamp'\n",
    "        if include_exif_data:\n",
    "            cmd += ' --include_exif_data'\n",
    "\n",
    "    cmd_file = os.path.join(filename_base,'run_chunk_{}_gpu_{}{}'.format(str(i_task).zfill(3),\n",
    "                            str(gpu_number).zfill(2),script_extension))\n",
    "\n",
    "    with open(cmd_file,'w') as f:\n",
    "        if script_header is not None and len(script_header) > 0:\n",
    "            f.write(script_header + '\\n')\n",
    "        f.write(cmd + '\\n')\n",
    "\n",
    "    st = os.stat(cmd_file)\n",
    "    os.chmod(cmd_file, st.st_mode | stat.S_IEXEC)\n",
    "\n",
    "    task['command'] = cmd\n",
    "    task['command_file'] = cmd_file\n",
    "\n",
    "    # Generate the script to resume from the checkpoint (only supported with MD inference code)\n",
    "\n",
    "    gpu_to_scripts[gpu_number].append(cmd_file)\n",
    "\n",
    "    if checkpoint_frequency is not None:\n",
    "\n",
    "        resume_string = ' --resume_from_checkpoint \"{}\"'.format(checkpoint_filename)\n",
    "        resume_cmd = cmd + resume_string\n",
    "\n",
    "        resume_cmd_file = os.path.join(filename_base,\n",
    "                                       'resume_chunk_{}_gpu_{}{}'.format(str(i_task).zfill(3),\n",
    "                                       str(gpu_number).zfill(2),script_extension))\n",
    "\n",
    "        with open(resume_cmd_file,'w') as f:\n",
    "            if script_header is not None and len(script_header) > 0:\n",
    "                f.write(script_header + '\\n')\n",
    "            f.write(resume_cmd + '\\n')\n",
    "\n",
    "        st = os.stat(resume_cmd_file)\n",
    "        os.chmod(resume_cmd_file, st.st_mode | stat.S_IEXEC)\n",
    "\n",
    "        task['resume_command'] = resume_cmd\n",
    "        task['resume_command_file'] = resume_cmd_file\n",
    "\n",
    "# ...for each task\n",
    "\n",
    "# Write out a script for each GPU that runs all of the commands associated with\n",
    "# that GPU.  Typically only used when running lots of little scripts in lieu\n",
    "# of checkpointing.\n",
    "for gpu_number in gpu_to_scripts:\n",
    "\n",
    "    gpu_script_file = os.path.join(filename_base,'run_all_for_gpu_{}{}'.format(\n",
    "        str(gpu_number).zfill(2),script_extension))\n",
    "    with open(gpu_script_file,'w') as f:\n",
    "        if script_header is not None and len(script_header) > 0:\n",
    "            f.write(script_header + '\\n')\n",
    "        for script_name in gpu_to_scripts[gpu_number]:\n",
    "            s = script_name\n",
    "            # When calling a series of batch files on Windows from within a batch file, you need to\n",
    "            # use \"call\", or only the first will be executed.  No, it doesn't make sense.\n",
    "            if os.name == 'nt':\n",
    "                s = 'call ' + s\n",
    "            f.write(s + '\\n')\n",
    "        f.write('echo \"Finished all commands for GPU {}\"'.format(gpu_number))\n",
    "    st = os.stat(gpu_script_file)\n",
    "    os.chmod(gpu_script_file, st.st_mode | stat.S_IEXEC)\n",
    "\n",
    "# ...for each GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d70dcc",
   "metadata": {},
   "source": [
    "## Run the tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc9ca3ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "r\"\"\"\n",
    "The cells we've run so far wrote out some shell scripts (.bat files on Windows,\n",
    ".sh files on Linx/Mac) that will run MegaDetector.  I like to leave the interactive\n",
    "environment at this point and run those scripts at the command line.  So, for example,\n",
    "if you're on Windows, and you've basically used the default values above, there will be\n",
    "batch files called, e.g.:\n",
    "\n",
    "c:\\users\\[username]\\postprocessing\\[organization]\\[job_name]\\run_chunk_000_gpu_00.bat\n",
    "c:\\users\\[username]\\postprocessing\\[organization]\\[job_name]\\run_chunk_001_gpu_01.bat\n",
    "\n",
    "Those batch files expect to be run from the \"detection\" folder of the MegaDetector repo,\n",
    "typically:\n",
    "\n",
    "c:\\git\\MegaDetector\\megadetector\\detection\n",
    "\n",
    "All of that said, you don't *have* to do this at the command line.  The following cell\n",
    "runs these scripts programmatically, so if you set \"run_tasks_in_notebook\" to \"True\"\n",
    "and run this cell, you can run MegaDetector without leaving this notebook.\n",
    "\n",
    "One downside of the programmatic approach is that this cell doesn't yet parallelize over\n",
    "multiple processes, so the tasks will run serially.  This only matters if you have\n",
    "multiple GPUs.\n",
    "\"\"\"\n",
    "\n",
    "run_tasks_in_notebook = False\n",
    "\n",
    "if run_tasks_in_notebook:\n",
    "\n",
    "    assert not use_yolo_inference_scripts, \\\n",
    "        'If you want to use the YOLOv5 inference scripts, you can\\'t run the model interactively (yet)'\n",
    "\n",
    "    # i_task = 0; task = task_info[i_task]\n",
    "    for i_task,task in enumerate(task_info):\n",
    "\n",
    "        chunk_file = task['input_file']\n",
    "        output_fn = task['output_file']\n",
    "\n",
    "        checkpoint_filename = chunk_file.replace('.json','_checkpoint.json')\n",
    "\n",
    "        if json_threshold is not None:\n",
    "            confidence_threshold = json_threshold\n",
    "        else:\n",
    "            confidence_threshold = DEFAULT_OUTPUT_CONFIDENCE_THRESHOLD\n",
    "\n",
    "        if checkpoint_frequency is not None and checkpoint_frequency > 0:\n",
    "            cp_freq_arg = checkpoint_frequency\n",
    "        else:\n",
    "            cp_freq_arg = -1\n",
    "\n",
    "        start_time = time.time()\n",
    "        results = load_and_run_detector_batch(model_file=model_file,\n",
    "                                              image_file_names=chunk_file,\n",
    "                                              checkpoint_path=checkpoint_filename,\n",
    "                                              confidence_threshold=confidence_threshold,\n",
    "                                              checkpoint_frequency=cp_freq_arg,\n",
    "                                              results=None,\n",
    "                                              n_cores=ncores,\n",
    "                                              use_image_queue=use_image_queue,\n",
    "                                              quiet=quiet_mode,\n",
    "                                              image_size=image_size)\n",
    "        elapsed = time.time() - start_time\n",
    "\n",
    "        print('Task {}: finished inference for {} images in {}'.format(\n",
    "            i_task, len(results),humanfriendly.format_timespan(elapsed)))\n",
    "\n",
    "        # This will write absolute paths to the file, we'll fix this later\n",
    "        write_results_to_file(results, output_fn, detector_file=model_file)\n",
    "\n",
    "        if checkpoint_frequency is not None and checkpoint_frequency > 0:\n",
    "            if os.path.isfile(checkpoint_filename):\n",
    "                os.remove(checkpoint_filename)\n",
    "                print('Deleted checkpoint file {}'.format(checkpoint_filename))\n",
    "\n",
    "    # ...for each chunk\n",
    "\n",
    "# ...if we're running tasks in this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f959dd4",
   "metadata": {},
   "source": [
    "## Load results, look for failed or missing images in each task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dd267ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that all task output files exist\n",
    "\n",
    "missing_output_files = []\n",
    "\n",
    "# i_task = 0; task = task_info[i_task]\n",
    "for i_task,task in tqdm(enumerate(task_info),total=len(task_info)):\n",
    "    output_file = task['output_file']\n",
    "    if not os.path.isfile(output_file):\n",
    "        missing_output_files.append(output_file)\n",
    "\n",
    "if len(missing_output_files) > 0:\n",
    "    print('Missing {} output files:'.format(len(missing_output_files)))\n",
    "    for s in missing_output_files:\n",
    "        print(s)\n",
    "    raise Exception('Missing output files')\n",
    "\n",
    "\n",
    "n_total_failures = 0\n",
    "\n",
    "# i_task = 0; task = task_info[i_task]\n",
    "for i_task,task in tqdm(enumerate(task_info),total=len(task_info)):\n",
    "\n",
    "    chunk_file = task['input_file']\n",
    "    output_file = task['output_file']\n",
    "\n",
    "    with open(chunk_file,'r') as f:\n",
    "        task_images = json.load(f)\n",
    "    with open(output_file,'r') as f:\n",
    "        task_results = json.load(f)\n",
    "\n",
    "    task_images_set = set(task_images)\n",
    "    filename_to_results = {}\n",
    "\n",
    "    n_task_failures = 0\n",
    "\n",
    "    # im = task_results['images'][0]\n",
    "    for im in task_results['images']:\n",
    "\n",
    "        # Most of the time, inference result files use absolute paths, but it's\n",
    "        # getting annoying to make sure that's *always* true, so handle both here.\n",
    "        # E.g., when using tiled inference, paths will be relative.\n",
    "        if not os.path.isabs(im['file']):\n",
    "            fn = os.path.join(input_path,im['file']).replace('\\\\','/')\n",
    "            im['file'] = fn\n",
    "        assert im['file'].startswith(input_path)\n",
    "        assert im['file'] in task_images_set\n",
    "        filename_to_results[im['file']] = im\n",
    "        if 'failure' in im:\n",
    "            assert im['failure'] is not None\n",
    "            n_task_failures += 1\n",
    "\n",
    "    task['n_failures'] = n_task_failures\n",
    "    task['results'] = task_results\n",
    "\n",
    "    for fn in task_images:\n",
    "        assert fn in filename_to_results, \\\n",
    "            'File {} not found in results for task {}'.format(fn,i_task)\n",
    "\n",
    "    n_total_failures += n_task_failures\n",
    "\n",
    "# ...for each task\n",
    "\n",
    "assert n_total_failures < max_tolerable_failed_images,\\\n",
    "    '{} failures (max tolerable set to {})'.format(n_total_failures,\n",
    "                                                   max_tolerable_failed_images)\n",
    "\n",
    "print('Processed all {} images with {} failures'.format(\n",
    "    len(all_images),n_total_failures))\n",
    "\n",
    "\n",
    "##%% Merge results files and make filenames relative\n",
    "\n",
    "combined_results = {}\n",
    "combined_results['images'] = []\n",
    "images_processed = set()\n",
    "\n",
    "for i_task,task in tqdm(enumerate(task_info),total=len(task_info)):\n",
    "\n",
    "    task_results = task['results']\n",
    "\n",
    "    if i_task == 0:\n",
    "        combined_results['info'] = task_results['info']\n",
    "        combined_results['detection_categories'] = task_results['detection_categories']\n",
    "    else:\n",
    "        assert task_results['info']['format_version'] == combined_results['info']['format_version']\n",
    "        assert task_results['detection_categories'] == combined_results['detection_categories']\n",
    "\n",
    "    # Make sure we didn't see this image in another chunk\n",
    "    for im in task_results['images']:\n",
    "        assert im['file'] not in images_processed\n",
    "        images_processed.add(im['file'])\n",
    "\n",
    "    combined_results['images'].extend(task_results['images'])\n",
    "\n",
    "# Check that we ended up with the right number of images\n",
    "assert len(combined_results['images']) == len(all_images), \\\n",
    "    'Expected {} images in combined results, found {}'.format(\n",
    "        len(all_images),len(combined_results['images']))\n",
    "\n",
    "# Check uniqueness\n",
    "result_filenames = [im['file'] for im in combined_results['images']]\n",
    "assert len(combined_results['images']) == len(set(result_filenames))\n",
    "\n",
    "# Convert to relative paths, preserving '/' as the path separator, regardless of OS\n",
    "for im in combined_results['images']:\n",
    "    assert '\\\\' not in im['file']\n",
    "    assert im['file'].startswith(input_path)\n",
    "    if input_path.endswith(':'):\n",
    "        im['file'] = im['file'].replace(input_path,'',1)\n",
    "    else:\n",
    "        im['file'] = im['file'].replace(input_path + '/','',1)\n",
    "\n",
    "with open(combined_api_output_file,'w') as f:\n",
    "    json.dump(combined_results,f,indent=1)\n",
    "\n",
    "print('Wrote results to {}'.format(combined_api_output_file))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6449c656",
   "metadata": {},
   "source": [
    "## Post-processing (pre-RDE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0267b58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "render_animals_only = False\n",
    "\n",
    "options = PostProcessingOptions()\n",
    "options.image_base_dir = input_path\n",
    "options.include_almost_detections = True\n",
    "options.num_images_to_sample = 7500\n",
    "options.confidence_threshold = 0.2\n",
    "options.almost_detection_confidence_threshold = options.confidence_threshold - 0.05\n",
    "options.ground_truth_json_file = None\n",
    "options.separate_detections_by_category = True\n",
    "options.sample_seed = 0\n",
    "options.max_figures_per_html_file = 2500\n",
    "\n",
    "options.parallelize_rendering = True\n",
    "options.parallelize_rendering_n_cores = default_workers_for_parallel_tasks\n",
    "options.parallelize_rendering_with_threads = parallelization_defaults_to_threads\n",
    "\n",
    "if render_animals_only:\n",
    "    # Omit some pages from the output, useful when animals are rare\n",
    "    options.rendering_bypass_sets = ['detections_person','detections_vehicle',\n",
    "                                     'detections_person_vehicle','non_detections']\n",
    "\n",
    "output_base = os.path.join(postprocessing_output_folder,\n",
    "    base_task_name + '_{:.3f}'.format(options.confidence_threshold))\n",
    "if render_animals_only:\n",
    "    output_base = output_base + '_animals_only'\n",
    "\n",
    "os.makedirs(output_base, exist_ok=True)\n",
    "print('Processing to {}'.format(output_base))\n",
    "\n",
    "options.md_results_file = combined_api_output_file\n",
    "options.output_dir = output_base\n",
    "ppresults = process_batch_results(options)\n",
    "html_output_file = ppresults.output_html_file\n",
    "path_utils.open_file(html_output_file,attempt_to_open_in_wsl_host=True,browser_name='chrome')\n",
    "# import clipboard; clipboard.copy(html_output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99de7baf",
   "metadata": {},
   "source": [
    "## Repeat detection elimination, phase 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "672c39e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deliberately leaving these imports here, rather than at the top, because this\n",
    "# cell is not typically executed\n",
    "from megadetector.postprocessing.repeat_detection_elimination import repeat_detections_core\n",
    "task_index = 0\n",
    "\n",
    "options = repeat_detections_core.RepeatDetectionOptions()\n",
    "\n",
    "options.confidenceMin = 0.1\n",
    "options.confidenceMax = 1.01\n",
    "options.iouThreshold = 0.85\n",
    "options.occurrenceThreshold = 15\n",
    "options.maxSuspiciousDetectionSize = 0.2\n",
    "# options.minSuspiciousDetectionSize = 0.05\n",
    "\n",
    "options.parallelizationUsesThreads = parallelization_defaults_to_threads\n",
    "options.nWorkers = default_workers_for_parallel_tasks\n",
    "\n",
    "# This will cause a very light gray box to get drawn around all the detections\n",
    "# we're *not* considering as suspicious.\n",
    "options.bRenderOtherDetections = True\n",
    "options.otherDetectionsThreshold = options.confidenceMin\n",
    "\n",
    "options.bRenderDetectionTiles = True\n",
    "options.maxOutputImageWidth = 2000\n",
    "options.detectionTilesMaxCrops = 250\n",
    "\n",
    "# options.lineThickness = 5\n",
    "# options.boxExpansion = 8\n",
    "\n",
    "# To invoke custom collapsing of folders for a particular manufacturer's naming scheme\n",
    "options.customDirNameFunction = relative_path_to_location\n",
    "\n",
    "options.bRenderHtml = False\n",
    "options.imageBase = input_path\n",
    "rde_string = 'rde_{:.3f}_{:.3f}_{}_{:.3f}'.format(\n",
    "    options.confidenceMin, options.iouThreshold,\n",
    "    options.occurrenceThreshold, options.maxSuspiciousDetectionSize)\n",
    "options.outputBase = os.path.join(filename_base, rde_string + '_task_{}'.format(task_index))\n",
    "options.filenameReplacements = None # {'':''}\n",
    "\n",
    "# Exclude people and vehicles from RDE\n",
    "# options.excludeClasses = [2,3]\n",
    "\n",
    "# options.maxImagesPerFolder = 50000\n",
    "# options.includeFolders = ['a/b/c']\n",
    "# options.excludeFolder = ['a/b/c']\n",
    "\n",
    "options.debugMaxDir = -1\n",
    "options.debugMaxRenderDir = -1\n",
    "options.debugMaxRenderDetection = -1\n",
    "options.debugMaxRenderInstance = -1\n",
    "\n",
    "# Can be None, 'xsort', or 'clustersort'\n",
    "options.smartSort = 'xsort'\n",
    "\n",
    "suspicious_detection_results = repeat_detections_core.find_repeat_detections(combined_api_output_file,\n",
    "                                                                             outputFilename=None,\n",
    "                                                                             options=options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f31ef7f3",
   "metadata": {},
   "source": [
    "## Manual RDE step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffa2336d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## DELETE THE VALID DETECTIONS ##\n",
    "\n",
    "# If you run this line, it will open the folder up in your file browser\n",
    "path_utils.open_file(os.path.dirname(suspicious_detection_results.filterFile),\n",
    "                     attempt_to_open_in_wsl_host=True)\n",
    "\n",
    "#\n",
    "# If you ran the previous cell, but then you change your mind and you don't want to do\n",
    "# the RDE step, that's fine, but don't just blast through this cell once you've run the\n",
    "# previous cell.  If you do that, you're implicitly telling the notebook that you looked\n",
    "# at everything in that folder, and confirmed there were no red boxes on animals.\n",
    "#\n",
    "# Instead, either change \"filtered_output_filename\" below to \"combined_api_output_file\",\n",
    "# or delete *all* the images in the filtering folder.\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0a7e73",
   "metadata": {},
   "source": [
    "## Re-filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1f53bf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from megadetector.postprocessing.repeat_detection_elimination import remove_repeat_detections\n",
    "\n",
    "filtered_output_filename = path_utils.insert_before_extension(combined_api_output_file,\n",
    "                                                              'filtered_{}'.format(rde_string))\n",
    "\n",
    "remove_repeat_detections.remove_repeat_detections(\n",
    "    inputFile=combined_api_output_file,\n",
    "    outputFile=filtered_output_filename,\n",
    "    filteringDir=os.path.dirname(suspicious_detection_results.filterFile)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43f5efe",
   "metadata": {},
   "source": [
    "## Post-processing (post-RDE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6dfc593",
   "metadata": {},
   "outputs": [],
   "source": [
    "render_animals_only = False\n",
    "\n",
    "options = PostProcessingOptions()\n",
    "options.image_base_dir = input_path\n",
    "options.include_almost_detections = True\n",
    "options.num_images_to_sample = 7500\n",
    "options.confidence_threshold = 0.2\n",
    "options.almost_detection_confidence_threshold = options.confidence_threshold - 0.05\n",
    "options.ground_truth_json_file = None\n",
    "options.separate_detections_by_category = True\n",
    "options.sample_seed = 0\n",
    "options.max_figures_per_html_file = 5000\n",
    "\n",
    "options.parallelize_rendering = True\n",
    "options.parallelize_rendering_n_cores = default_workers_for_parallel_tasks\n",
    "options.parallelize_rendering_with_threads = parallelization_defaults_to_threads\n",
    "\n",
    "if render_animals_only:\n",
    "    # Omit some pages from the output, useful when animals are rare\n",
    "    options.rendering_bypass_sets = ['detections_person','detections_vehicle',\n",
    "                                      'detections_person_vehicle','non_detections']\n",
    "\n",
    "output_base = os.path.join(postprocessing_output_folder,\n",
    "    base_task_name + '_{}_{:.3f}'.format(rde_string, options.confidence_threshold))\n",
    "\n",
    "if render_animals_only:\n",
    "    output_base = output_base + '_render_animals_only'\n",
    "os.makedirs(output_base, exist_ok=True)\n",
    "\n",
    "print('Processing post-RDE to {}'.format(output_base))\n",
    "\n",
    "options.md_results_file = filtered_output_filename\n",
    "options.output_dir = output_base\n",
    "ppresults = process_batch_results(options)\n",
    "html_output_file = ppresults.output_html_file\n",
    "\n",
    "path_utils.open_file(html_output_file,attempt_to_open_in_wsl_host=True,browser_name='chrome')\n",
    "# import clipboard; clipboard.copy(html_output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958a96e0",
   "metadata": {},
   "source": [
    "## Run MegaClassifier (actually, write out a script that runs MegaClassifier)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4500b87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variables that will indicate which classifiers we ran\n",
    "final_output_path_mc = None\n",
    "final_output_path_ic = None\n",
    "\n",
    "# If we didn't do RDE\n",
    "if filtered_output_filename is None:\n",
    "    print(\"Warning: it looks like you didn't do RDE, using the raw output file\")\n",
    "    filtered_output_filename = combined_api_output_file\n",
    "\n",
    "classifier_name_short = 'megaclassifier'\n",
    "threshold_str = '0.15' # 0.6\n",
    "classifier_name = 'megaclassifier_v0.1_efficientnet-b3'\n",
    "\n",
    "organization_name = organization_name_short\n",
    "job_name = base_task_name\n",
    "input_filename = filtered_output_filename # combined_api_output_file\n",
    "input_files = [input_filename]\n",
    "image_base = input_path\n",
    "crop_path = os.path.join(os.path.expanduser('~/crops'),job_name + '_crops')\n",
    "output_base = combined_api_output_folder\n",
    "device_id = 0\n",
    "\n",
    "output_file = os.path.join(filename_base,'run_{}_'.format(classifier_name_short) + job_name + script_extension)\n",
    "\n",
    "classifier_base = os.path.expanduser('~/models/camera_traps/megaclassifier/v0.1/')\n",
    "assert os.path.isdir(classifier_base)\n",
    "\n",
    "checkpoint_path = os.path.join(classifier_base,'v0.1_efficientnet-b3_compiled.pt')\n",
    "assert os.path.isfile(checkpoint_path)\n",
    "\n",
    "classifier_categories_path = os.path.join(classifier_base,'v0.1_index_to_name.json')\n",
    "assert os.path.isfile(classifier_categories_path)\n",
    "\n",
    "target_mapping_path = os.path.join(classifier_base,'idfg_to_megaclassifier_labels.json')\n",
    "assert os.path.isfile(target_mapping_path)\n",
    "\n",
    "classifier_output_suffix = '_megaclassifier_output.csv.gz'\n",
    "final_output_suffix = '_megaclassifier.json'\n",
    "\n",
    "n_threads_str = str(default_workers_for_parallel_tasks)\n",
    "image_size_str = '300'\n",
    "batch_size_str = '64'\n",
    "num_workers_str = str(default_workers_for_parallel_tasks)\n",
    "classification_threshold_str = '0.05'\n",
    "\n",
    "logdir = filename_base\n",
    "\n",
    "# This is just passed along to the metadata in the output file, it has no impact\n",
    "# on how the classification scripts run.\n",
    "typical_classification_threshold_str = '0.75'\n",
    "\n",
    "\n",
    "##%% Set up environment\n",
    "\n",
    "commands = []\n",
    "# commands.append('cd MegaDetector/megadetector/classification\\n')\n",
    "# commands.append('mamba activate cameratraps-classifier\\n')\n",
    "\n",
    "if script_header is not None and len(script_header) > 0:\n",
    "    commands.append(script_header)\n",
    "\n",
    "\n",
    "##%% Crop images\n",
    "\n",
    "commands.append('\\n' + scc + ' Cropping ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    crop_cmd = ''\n",
    "\n",
    "    crop_comment = '\\n' + scc + ' Cropping {}\\n\\n'.format(fn)\n",
    "    crop_cmd += crop_comment\n",
    "\n",
    "    crop_cmd += \"python crop_detections.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + input_file_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + crop_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--images-dir \"' + image_base + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threshold \"' + threshold_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--square-crops ' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threads \"' + n_threads_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--logdir \"' + logdir + '\"' + '\\n' + \\\n",
    "         ' ' + '\\n'\n",
    "    crop_cmd = '{}'.format(crop_cmd)\n",
    "    commands.append(crop_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Run classifier\n",
    "\n",
    "commands.append('\\n' + scc + ' Classifying ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    classifier_output_path = crop_path + classifier_output_suffix\n",
    "\n",
    "    classify_cmd = ''\n",
    "\n",
    "    classify_comment = '\\n' + scc + ' Classifying {}\\n\\n'.format(fn)\n",
    "    classify_cmd += classify_comment\n",
    "\n",
    "    classify_cmd += \"python run_classifier.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + checkpoint_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + crop_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + classifier_output_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--detections-json \"' + input_file_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--classifier-categories \"' + classifier_categories_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--image-size \"' + image_size_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--batch-size \"' + batch_size_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--num-workers \"' + num_workers_str + '\"' + ' ' + slcc + '\\n'\n",
    "\n",
    "    if device_id is not None:\n",
    "        classify_cmd += ' ' + '--device {}'.format(device_id)\n",
    "\n",
    "    classify_cmd += '\\n\\n'\n",
    "    classify_cmd = '{}'.format(classify_cmd)\n",
    "    commands.append(classify_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Remap classifier outputs\n",
    "\n",
    "commands.append('\\n' + scc + ' Remapping ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    classifier_output_path = crop_path + classifier_output_suffix\n",
    "    classifier_output_path_remapped = \\\n",
    "        classifier_output_path.replace(\".csv.gz\",\"_remapped.csv.gz\")\n",
    "    assert not (classifier_output_path == classifier_output_path_remapped)\n",
    "\n",
    "    output_label_index = classifier_output_path_remapped.replace(\n",
    "        \"_remapped.csv.gz\",\"_label_index_remapped.json\")\n",
    "\n",
    "    remap_cmd = ''\n",
    "\n",
    "    remap_comment = '\\n' + scc + ' Remapping {}\\n\\n'.format(fn)\n",
    "    remap_cmd += remap_comment\n",
    "\n",
    "    remap_cmd += \"python aggregate_classifier_probs.py \" + slcc + \"\\n\" + \\\n",
    "        ' \"' + classifier_output_path + '\" ' + slcc + '\\n' + \\\n",
    "        ' ' + '--target-mapping \"' + target_mapping_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "        ' ' + '--output-csv \"' + classifier_output_path_remapped + '\"' + ' ' + slcc + '\\n' + \\\n",
    "        ' ' + '--output-label-index \"' + output_label_index + '\"' \\\n",
    "        '\\n'\n",
    "\n",
    "    remap_cmd = '{}'.format(remap_cmd)\n",
    "    commands.append(remap_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Merge classification and detection outputs\n",
    "\n",
    "commands.append('\\n\\n' + scc + ' Merging ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    classifier_output_path = crop_path + classifier_output_suffix\n",
    "\n",
    "    classifier_output_path_remapped = \\\n",
    "        classifier_output_path.replace(\".csv.gz\",\"_remapped.csv.gz\")\n",
    "\n",
    "    output_label_index = classifier_output_path_remapped.replace(\n",
    "        \"_remapped.csv.gz\",\"_label_index_remapped.json\")\n",
    "\n",
    "    final_output_path = os.path.join(output_base,\n",
    "                                     os.path.basename(classifier_output_path)).\\\n",
    "        replace(classifier_output_suffix,\n",
    "        final_output_suffix)\n",
    "    final_output_path = final_output_path.replace('_detections','')\n",
    "    final_output_path = final_output_path.replace('_crops','')\n",
    "    final_output_path_mc = final_output_path\n",
    "\n",
    "    merge_cmd = ''\n",
    "\n",
    "    merge_comment = '\\n' + scc + ' Merging {}\\n\\n'.format(fn)\n",
    "    merge_cmd += merge_comment\n",
    "\n",
    "    merge_cmd += \"python merge_classification_detection_output.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + classifier_output_path_remapped + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + output_label_index + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--output-json \"' + final_output_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--detection-json \"' + input_file_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--classifier-name \"' + classifier_name + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threshold \"' + classification_threshold_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--typical-confidence-threshold \"' + typical_classification_threshold_str + '\"' + '\\n' + \\\n",
    "         '\\n'\n",
    "    merge_cmd = '{}'.format(merge_cmd)\n",
    "    commands.append(merge_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Write out classification script\n",
    "\n",
    "with open(output_file,'w') as f:\n",
    "    for s in commands:\n",
    "        f.write('{}'.format(s))\n",
    "\n",
    "st = os.stat(output_file)\n",
    "os.chmod(output_file, st.st_mode | stat.S_IEXEC)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24e0b55c",
   "metadata": {},
   "source": [
    "## Run a non-MegaClassifier classifier (i.e., a classifier with no output mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e2a002a",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier_name_short = 'idfgclassifier'\n",
    "threshold_str = '0.15' # 0.6\n",
    "classifier_name = 'idfg_classifier_ckpt_14_compiled'\n",
    "\n",
    "organization_name = organization_name_short\n",
    "job_name = base_task_name\n",
    "input_filename = filtered_output_filename # combined_api_output_file\n",
    "input_files = [input_filename]\n",
    "image_base = input_path\n",
    "crop_path = os.path.join(os.path.expanduser('~/crops'),job_name + '_crops')\n",
    "output_base = combined_api_output_folder\n",
    "device_id = 1\n",
    "\n",
    "output_file = os.path.join(filename_base,'run_{}_'.format(classifier_name_short) + job_name +  script_extension)\n",
    "\n",
    "classifier_base = os.path.expanduser('~/models/camera_traps/idfg_classifier/idfg_classifier_20200905_042558')\n",
    "assert os.path.isdir(classifier_base)\n",
    "\n",
    "checkpoint_path = os.path.join(classifier_base,'idfg_classifier_ckpt_14_compiled.pt')\n",
    "assert os.path.isfile(checkpoint_path)\n",
    "\n",
    "classifier_categories_path = os.path.join(classifier_base,'label_index.json')\n",
    "assert os.path.isfile(classifier_categories_path)\n",
    "\n",
    "classifier_output_suffix = '_{}_output.csv.gz'.format(classifier_name_short)\n",
    "final_output_suffix = '_{}.json'.format(classifier_name_short)\n",
    "\n",
    "threshold_str = '0.65'\n",
    "n_threads_str = str(default_workers_for_parallel_tasks)\n",
    "image_size_str = '300'\n",
    "batch_size_str = '64'\n",
    "num_workers_str = str(default_workers_for_parallel_tasks)\n",
    "logdir = filename_base\n",
    "\n",
    "classification_threshold_str = '0.05'\n",
    "\n",
    "# This is just passed along to the metadata in the output file, it has no impact\n",
    "# on how the classification scripts run.\n",
    "typical_classification_threshold_str = '0.75'\n",
    "\n",
    "\n",
    "##%% Set up environment\n",
    "\n",
    "commands = []\n",
    "if script_header is not None and len(script_header) > 0:\n",
    "    commands.append(script_header)\n",
    "\n",
    "\n",
    "##%% Crop images\n",
    "\n",
    "commands.append('\\n' + scc + ' Cropping ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    crop_cmd = ''\n",
    "\n",
    "    crop_comment = '\\n' + scc + ' Cropping {}\\n'.format(fn)\n",
    "    crop_cmd += crop_comment\n",
    "\n",
    "    crop_cmd += \"python crop_detections.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + input_file_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + crop_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--images-dir \"' + image_base + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threshold \"' + threshold_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--square-crops ' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threads \"' + n_threads_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--logdir \"' + logdir + '\"' + '\\n' + \\\n",
    "         '\\n'\n",
    "    crop_cmd = '{}'.format(crop_cmd)\n",
    "    commands.append(crop_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Run classifier\n",
    "\n",
    "commands.append('\\n' + scc + ' Classifying ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    classifier_output_path = crop_path + classifier_output_suffix\n",
    "\n",
    "    classify_cmd = ''\n",
    "\n",
    "    classify_comment = '\\n' + scc + ' Classifying {}\\n'.format(fn)\n",
    "    classify_cmd += classify_comment\n",
    "\n",
    "    classify_cmd += \"python run_classifier.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + checkpoint_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + crop_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + classifier_output_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--detections-json \"' + input_file_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--classifier-categories \"' + classifier_categories_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--image-size \"' + image_size_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--batch-size \"' + batch_size_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--num-workers \"' + num_workers_str + '\"' + ' ' + slcc + '\\n'\n",
    "\n",
    "    if device_id is not None:\n",
    "        classify_cmd += ' ' + '--device {}'.format(device_id)\n",
    "\n",
    "    classify_cmd += '\\n\\n'\n",
    "    classify_cmd = '{}'.format(classify_cmd)\n",
    "    commands.append(classify_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Merge classification and detection outputs\n",
    "\n",
    "commands.append('\\n' + scc + ' Merging ' + scc + '\\n')\n",
    "\n",
    "# fn = input_files[0]\n",
    "for fn in input_files:\n",
    "\n",
    "    input_file_path = fn\n",
    "    classifier_output_path = crop_path + classifier_output_suffix\n",
    "    final_output_path = os.path.join(output_base,\n",
    "                                     os.path.basename(classifier_output_path)).\\\n",
    "                                     replace(classifier_output_suffix,\n",
    "                                     final_output_suffix)\n",
    "    final_output_path = final_output_path.replace('_detections','')\n",
    "    final_output_path = final_output_path.replace('_crops','')\n",
    "    final_output_path_ic = final_output_path\n",
    "\n",
    "    merge_cmd = ''\n",
    "\n",
    "    merge_comment = '\\n' + scc + ' Merging {}\\n'.format(fn)\n",
    "    merge_cmd += merge_comment\n",
    "\n",
    "    merge_cmd += \"python merge_classification_detection_output.py \" + slcc + \"\\n\" + \\\n",
    "    \t ' \"' + classifier_output_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' \"' + classifier_categories_path + '\" ' + slcc + '\\n' + \\\n",
    "         ' ' + '--output-json \"' + final_output_path_ic + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--detection-json \"' + input_file_path + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--classifier-name \"' + classifier_name + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--threshold \"' + classification_threshold_str + '\"' + ' ' + slcc + '\\n' + \\\n",
    "         ' ' + '--typical-confidence-threshold \"' + typical_classification_threshold_str + '\"' + '\\n' + \\\n",
    "         '\\n'\n",
    "    merge_cmd = '{}'.format(merge_cmd)\n",
    "    commands.append(merge_cmd)\n",
    "\n",
    "    if len(command_suffix) > 0:\n",
    "        commands.append(command_suffix)\n",
    "\n",
    "\n",
    "##%% Write everything out\n",
    "\n",
    "with open(output_file,'w') as f:\n",
    "    for s in commands:\n",
    "        f.write('{}'.format(s))\n",
    "\n",
    "import stat\n",
    "st = os.stat(output_file)\n",
    "os.chmod(output_file, st.st_mode | stat.S_IEXEC)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a33ae99",
   "metadata": {},
   "source": [
    "## Run the classifier(s) via the .sh script(s) or batch file(s) we just wrote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09d698d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# I do this manually, primarily because this requires a different mamba environment\n",
    "# (cameratraps-classifier) from MegaDetector's environment (cameratraps-detector).\n",
    "#\n",
    "# The next few pseudo-cells (#%) in this script are basically always run all at once, getting us\n",
    "# all the way from running the classifier to classification previews and zipped .json files that\n",
    "# are ready to upload.\n",
    "\n",
    "\n",
    "##%% Do All The Rest of The Stuff\n",
    "\n",
    "# The remaining cells require no human intervention, so although a few conceptually-unrelated things\n",
    "# happen (e.g., sequence-level smoothing, preview generation, and zipping all the .json files), I group these\n",
    "# all into one super-cell.  Every click counts.\n",
    "\n",
    "\n",
    "##%% Within-image classification smoothing\n",
    "\n",
    "classification_detection_files = []\n",
    "\n",
    "from megadetector.postprocessing.classification_postprocessing import \\\n",
    "    smooth_classification_results_image_level\n",
    "\n",
    "# Did we run MegaClassifier?\n",
    "if final_output_path_mc is not None:\n",
    "    classification_detection_files.append(final_output_path_mc)\n",
    "\n",
    "# Did we run the IDFG classifier?\n",
    "if final_output_path_ic is not None:\n",
    "    classification_detection_files.append(final_output_path_ic)\n",
    "\n",
    "assert all([os.path.isfile(fn) for fn in classification_detection_files])\n",
    "\n",
    "smoothed_classification_files = []\n",
    "\n",
    "for final_output_path in classification_detection_files:\n",
    "\n",
    "    classifier_output_path = final_output_path\n",
    "    classifier_output_path_within_image_smoothing = classifier_output_path.replace(\n",
    "        '.json','_within_image_smoothing.json')\n",
    "    smoothed_classification_files.append(classifier_output_path_within_image_smoothing)\n",
    "    smooth_classification_results_image_level(input_file=classifier_output_path,\n",
    "                                              output_file=classifier_output_path_within_image_smoothing,\n",
    "                                              options=None)\n",
    "\n",
    "# ...for each file we want to smooth\n",
    "\n",
    "\n",
    "##%% Read EXIF date and time from all images\n",
    "\n",
    "from megadetector.data_management import read_exif\n",
    "exif_options = read_exif.ReadExifOptions()\n",
    "\n",
    "exif_options.verbose = False\n",
    "exif_options.n_workers = default_workers_for_parallel_tasks\n",
    "exif_options.use_threads = parallelization_defaults_to_threads\n",
    "exif_options.processing_library = 'pil'\n",
    "exif_options.byte_handling = 'delete'\n",
    "exif_options.tags_to_include = ['DateTime','DateTimeOriginal']\n",
    "\n",
    "exif_results_file = os.path.join(filename_base,'exif_data.json')\n",
    "\n",
    "if os.path.isfile(exif_results_file):\n",
    "    print('Reading EXIF results from {}'.format(exif_results_file))\n",
    "    with open(exif_results_file,'r') as f:\n",
    "        exif_results = json.load(f)\n",
    "else:\n",
    "    exif_results = read_exif.read_exif_from_folder(input_path,\n",
    "                                                   output_file=exif_results_file,\n",
    "                                                   options=exif_options)\n",
    "\n",
    "\n",
    "##%% Prepare COCO-camera-traps-compatible image objects for EXIF results\n",
    "\n",
    "# ...and add location/datetime info based on filenames and EXIF information.\n",
    "\n",
    "from megadetector.data_management.read_exif import exif_results_to_cct\n",
    "\n",
    "cct_dict = exif_results_to_cct(exif_results=exif_results,\n",
    "                               cct_output_file=None,\n",
    "                               options=None)\n",
    "\n",
    "\n",
    "##%% Assemble images into sequences\n",
    "\n",
    "from megadetector.data_management import cct_json_utils\n",
    "\n",
    "print('Assembling images into sequences')\n",
    "cct_json_utils.create_sequences(cct_dict)\n",
    "\n",
    "\n",
    "##%% Sequence-level smoothing\n",
    "\n",
    "from megadetector.postprocessing.classification_postprocessing import \\\n",
    "    ClassificationSmoothingOptionsSequenceLevel, smooth_classification_results_sequence_level\n",
    "\n",
    "options = ClassificationSmoothingOptionsSequenceLevel()\n",
    "options.category_names_to_smooth_to = set(['deer','elk','cow','canid','cat','bird','bear'])\n",
    "options.min_dominant_class_ratio_for_secondary_override_table = {'cow':2,None:3}\n",
    "\n",
    "sequence_level_smoothing_input_file = smoothed_classification_files[0]\n",
    "sequence_smoothed_classification_file = sequence_level_smoothing_input_file.replace(\n",
    "    '.json','_seqsmoothing.json')\n",
    "\n",
    "_ = smooth_classification_results_sequence_level(md_results=sequence_level_smoothing_input_file,\n",
    "                                             cct_sequence_information=cct_dict,\n",
    "                                             output_file=sequence_smoothed_classification_file,\n",
    "                                             options=options)\n",
    "\n",
    "\n",
    "##%% Post-processing (post-classification, post-within-image-and-within-sequence-smoothing)\n",
    "\n",
    "options = PostProcessingOptions()\n",
    "options.image_base_dir = input_path\n",
    "options.include_almost_detections = True\n",
    "options.num_images_to_sample = 10000\n",
    "options.confidence_threshold = 0.2\n",
    "options.classification_confidence_threshold = 0.7\n",
    "options.almost_detection_confidence_threshold = options.confidence_threshold - 0.05\n",
    "options.ground_truth_json_file = None\n",
    "options.separate_detections_by_category = True\n",
    "options.max_figures_per_html_file = 2500\n",
    "\n",
    "options.parallelize_rendering = True\n",
    "options.parallelize_rendering_n_cores = default_workers_for_parallel_tasks\n",
    "options.parallelize_rendering_with_threads = parallelization_defaults_to_threads\n",
    "\n",
    "folder_token = sequence_smoothed_classification_file.split(os.path.sep)[-1].replace(\n",
    "    '_within_image_smoothing_seqsmoothing','')\n",
    "folder_token = folder_token.replace('.json','_seqsmoothing')\n",
    "\n",
    "output_base = os.path.join(postprocessing_output_folder, folder_token + \\\n",
    "    base_task_name + '_{:.3f}'.format(options.confidence_threshold))\n",
    "os.makedirs(output_base, exist_ok=True)\n",
    "print('Processing {} to {}'.format(base_task_name, output_base))\n",
    "\n",
    "options.md_results_file = sequence_smoothed_classification_file\n",
    "options.output_dir = output_base\n",
    "ppresults = process_batch_results(options)\n",
    "path_utils.open_file(ppresults.output_html_file,attempt_to_open_in_wsl_host=True,browser_name='chrome')\n",
    "# import clipboard; clipboard.copy(ppresults.output_html_file)\n",
    "\n",
    "\n",
    "##%% Zip .json files\n",
    "\n",
    "from megadetector.utils.path_utils import parallel_zip_files\n",
    "\n",
    "json_files = os.listdir(combined_api_output_folder)\n",
    "json_files = [fn for fn in json_files if fn.endswith('.json')]\n",
    "json_files = [os.path.join(combined_api_output_folder,fn) for fn in json_files]\n",
    "\n",
    "parallel_zip_files(json_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16dc31c5",
   "metadata": {},
   "source": [
    "## 99.9% of jobs end here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62105f60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Everything after this is run ad hoc and/or requires some manual editing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "091851ad",
   "metadata": {},
   "source": [
    "## Compare results files for different model versions (or before/after RDE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6d932e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "from megadetector.postprocessing.compare_batch_results import \\\n",
    "    BatchComparisonOptions, PairwiseBatchComparisonOptions, compare_batch_results\n",
    "\n",
    "options = BatchComparisonOptions()\n",
    "\n",
    "options.job_name = organization_name_short\n",
    "options.output_folder = os.path.join(postprocessing_output_folder,'model_comparison')\n",
    "options.image_folder = input_path\n",
    "\n",
    "options.pairwise_options = []\n",
    "\n",
    "filenames = [\n",
    "    '/postprocessing/organization/mdv4_results.json',\n",
    "    '/postprocessing/organization/mdv5a_results.json',\n",
    "    '/postprocessing/organization/mdv5b_results.json'\n",
    "    ]\n",
    "\n",
    "detection_thresholds = [0.7,0.15,0.15]\n",
    "\n",
    "assert len(detection_thresholds) == len(filenames)\n",
    "\n",
    "rendering_thresholds = [(x*0.6666) for x in detection_thresholds]\n",
    "\n",
    "# Choose all pairwise combinations of the files in [filenames]\n",
    "for i, j in itertools.combinations(list(range(0,len(filenames))),2):\n",
    "\n",
    "    pairwise_options = PairwiseBatchComparisonOptions()\n",
    "\n",
    "    pairwise_options.results_filename_a = filenames[i]\n",
    "    pairwise_options.results_filename_b = filenames[j]\n",
    "\n",
    "    pairwise_options.rendering_confidence_threshold_a = rendering_thresholds[i]\n",
    "    pairwise_options.rendering_confidence_threshold_b = rendering_thresholds[j]\n",
    "\n",
    "    pairwise_options.detection_thresholds_a = {'animal':detection_thresholds[i],\n",
    "                                               'person':detection_thresholds[i],\n",
    "                                               'vehicle':detection_thresholds[i]}\n",
    "    pairwise_options.detection_thresholds_b = {'animal':detection_thresholds[j],\n",
    "                                               'person':detection_thresholds[j],\n",
    "                                               'vehicle':detection_thresholds[j]}\n",
    "    options.pairwise_options.append(pairwise_options)\n",
    "\n",
    "results = compare_batch_results(options)\n",
    "\n",
    "from megadetector.utils.path_utils import open_file\n",
    "open_file(results.html_output_file,attempt_to_open_in_wsl_host=True,browser_name='chrome')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990048e0",
   "metadata": {},
   "source": [
    "## Merge in high-confidence detections from another results file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a2ed40a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from megadetector.postprocessing.merge_detections import \\\n",
    "    MergeDetectionsOptions,merge_detections\n",
    "\n",
    "source_files = ['']\n",
    "target_file = ''\n",
    "output_file = target_file.replace('.json','_merged.json')\n",
    "\n",
    "options = MergeDetectionsOptions()\n",
    "options.max_detection_size = 1.0\n",
    "options.target_confidence_threshold = 0.25\n",
    "options.categories_to_include = [1]\n",
    "options.source_confidence_thresholds = [0.2]\n",
    "merge_detections(source_files, target_file, output_file, options)\n",
    "\n",
    "merged_detections_file = output_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aca1980",
   "metadata": {},
   "source": [
    "## Create a new category for large boxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b79aa3c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from megadetector.postprocessing import categorize_detections_by_size\n",
    "\n",
    "size_options = categorize_detections_by_size.SizeCategorizationOptions()\n",
    "\n",
    "size_options.size_thresholds = [0.9]\n",
    "size_options.size_category_names = ['large_detections']\n",
    "\n",
    "size_options.categories_to_separate = [1]\n",
    "size_options.measurement = 'size' # 'width'\n",
    "\n",
    "threshold_string = '-'.join([str(x) for x in size_options.size_thresholds])\n",
    "\n",
    "input_file = filtered_output_filename\n",
    "size_separated_file = input_file.replace('.json','-size-separated-{}.json'.format(\n",
    "    threshold_string))\n",
    "d = categorize_detections_by_size.categorize_detections_by_size(input_file,size_separated_file,\n",
    "                                                                size_options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405862c4",
   "metadata": {},
   "source": [
    "## Preview large boxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "572a356c",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_base_large_boxes = os.path.join(postprocessing_output_folder,\n",
    "    base_task_name + '_{}_{:.3f}_size_separated_boxes'.format(rde_string, options.confidence_threshold))\n",
    "os.makedirs(output_base_large_boxes, exist_ok=True)\n",
    "print('Processing post-RDE, post-size-separation to {}'.format(output_base_large_boxes))\n",
    "\n",
    "options.md_results_file = size_separated_file\n",
    "options.output_dir = output_base_large_boxes\n",
    "\n",
    "ppresults = process_batch_results(options)\n",
    "html_output_file = ppresults.output_html_file\n",
    "path_utils.open_file(html_output_file,attempt_to_open_in_wsl_host=True,browser_name='chrome')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7207d133",
   "metadata": {},
   "source": [
    "## .json splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9793592a",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = None\n",
    "\n",
    "from megadetector.postprocessing.subset_json_detector_output import \\\n",
    "    subset_json_detector_output, SubsetJsonDetectorOutputOptions\n",
    "\n",
    "input_filename = filtered_output_filename\n",
    "output_base = os.path.join(combined_api_output_folder,base_task_name + '_json_subsets')\n",
    "\n",
    "print('Processing file {} to {}'.format(input_filename,output_base))\n",
    "\n",
    "options = SubsetJsonDetectorOutputOptions()\n",
    "# options.query = None\n",
    "# options.replacement = None\n",
    "\n",
    "options.split_folders = True\n",
    "options.make_folder_relative = True\n",
    "\n",
    "# Reminder: 'n_from_bottom' with a parameter of zero is the same as 'bottom'\n",
    "options.split_folder_mode = 'bottom'  # 'top', 'n_from_top', 'n_from_bottom'\n",
    "options.split_folder_param = 0\n",
    "options.overwrite_json_files = False\n",
    "options.confidence_threshold = 0.01\n",
    "\n",
    "subset_data = subset_json_detector_output(input_filename, output_base, options, data)\n",
    "\n",
    "# Zip the subsets folder\n",
    "from megadetector.utils.path_utils import zip_folder\n",
    "zip_folder(output_base,verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b565486",
   "metadata": {},
   "source": [
    "## Custom splitting/subsetting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc243957",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = None\n",
    "\n",
    "from megadetector.postprocessing.subset_json_detector_output import \\\n",
    "    subset_json_detector_output, SubsetJsonDetectorOutputOptions\n",
    "\n",
    "input_filename = filtered_output_filename\n",
    "output_base = os.path.join(filename_base,'json_subsets')\n",
    "\n",
    "folders = os.listdir(input_path)\n",
    "\n",
    "if data is None:\n",
    "    with open(input_filename) as f:\n",
    "        data = json.load(f)\n",
    "\n",
    "print('Data set contains {} images'.format(len(data['images'])))\n",
    "\n",
    "# i_folder = 0; folder_name = folders[i_folder]\n",
    "for i_folder, folder_name in enumerate(folders):\n",
    "\n",
    "    output_filename = os.path.join(output_base, folder_name + '.json')\n",
    "    print('Processing folder {} of {} ({}) to {}'.format(i_folder, len(folders), folder_name,\n",
    "          output_filename))\n",
    "\n",
    "    options = SubsetJsonDetectorOutputOptions()\n",
    "    options.confidence_threshold = 0.01\n",
    "    options.overwrite_json_files = True\n",
    "    options.query = folder_name + '/'\n",
    "\n",
    "    # This doesn't do anything in this case, since we're not splitting folders\n",
    "    # options.make_folder_relative = True\n",
    "\n",
    "    subset_data = subset_json_detector_output(input_filename, output_filename, options, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b928b77",
   "metadata": {},
   "source": [
    "## String replacement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f82112d",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = None\n",
    "\n",
    "from megadetector.postprocessing.subset_json_detector_output import \\\n",
    "    subset_json_detector_output, SubsetJsonDetectorOutputOptions\n",
    "\n",
    "input_filename = filtered_output_filename\n",
    "output_filename = input_filename.replace('.json','_replaced.json')\n",
    "\n",
    "options = SubsetJsonDetectorOutputOptions()\n",
    "options.query = folder_name + '/'\n",
    "options.replacement = ''\n",
    "subset_json_detector_output(input_filename,output_filename,options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77808dbb",
   "metadata": {},
   "source": [
    "## Splitting images into folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14e1c797",
   "metadata": {},
   "outputs": [],
   "source": [
    "from megadetector.postprocessing.separate_detections_into_folders import \\\n",
    "    separate_detections_into_folders, SeparateDetectionsIntoFoldersOptions\n",
    "\n",
    "default_threshold = 0.2\n",
    "base_output_folder = os.path.expanduser('~/data/{}-{}-separated'.format(base_task_name,default_threshold))\n",
    "\n",
    "options = SeparateDetectionsIntoFoldersOptions(default_threshold)\n",
    "\n",
    "options.results_file = filtered_output_filename\n",
    "options.base_input_folder = input_path\n",
    "options.base_output_folder = os.path.join(base_output_folder,folder_name)\n",
    "options.n_threads = default_workers_for_parallel_tasks\n",
    "options.allow_existing_directory = False\n",
    "\n",
    "separate_detections_into_folders(options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f2ef80f",
   "metadata": {},
   "source": [
    "## Convert frame-level results to video-level results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "449b5066",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This cell is only useful if the files submitted to this job were generated via\n",
    "# video_folder_to_frames().\n",
    "\n",
    "from megadetector.detection.video_utils import frame_results_to_video_results\n",
    "\n",
    "video_output_filename = filtered_output_filename.replace('.json','_aggregated.json')\n",
    "frame_results_to_video_results(filtered_output_filename,video_output_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11a8d906",
   "metadata": {},
   "source": [
    "## Sample custom path replacement function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abd1dde8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def custom_relative_path_to_location(relative_path):\n",
    "\n",
    "    relative_path = relative_path.replace('\\\\','/')\n",
    "    tokens = relative_path.split('/')\n",
    "    location_name = '/'.join(tokens[0:2])\n",
    "    return location_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79dd4e1b",
   "metadata": {},
   "source": [
    "## Test relative_path_to_location on the current dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d835e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(combined_api_output_file,'r') as f:\n",
    "    d = json.load(f)\n",
    "image_filenames = [im['file'] for im in d['images']]\n",
    "\n",
    "location_names = set()\n",
    "\n",
    "# relative_path = image_filenames[0]\n",
    "for relative_path in tqdm(image_filenames):\n",
    "    location_name = relative_path_to_location(relative_path)\n",
    "    location_names.add(location_name)\n",
    "\n",
    "location_names = list(location_names)\n",
    "location_names.sort()\n",
    "\n",
    "for s in location_names:\n",
    "    print(s)"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}
